How to Stop Search Engine from Crawling Your WordPress Website

June 4th, 2017 | by Ravi Chahar || 4 Comments |

Do you want to stop search engine from crawling your WordPress website? It may sound odd to many. People may think why would anyone doesn’t want the search engine to crawl his/her website.

Well, if you’re a web developer or a web designer, you can easily get this point. While running a WordPress website, you may in need to make some changes, you may even want to change the domain name.

There are many people who work on the live websites instead of creating a local environment.

Many people even host their projects online to build their portfolio so that more potential clients can be attracted. So they don’t need to index those websites.

In this article, you will learn the different ways to stop search engine from crawling your WordPress website.

Use the WordPress Admin Panel to Discourage the Search Engine from Crawling and indexing

If you have a little bit of knowledge about WordPress then you may have seen a built-in feature which can do the work.

Go to Settings>>Reading and when you scroll down, you will see a checkbox. If you check this box, the search engine will stop indexing your WordPress website.

Save the settings and you’re all set.

It adds a meta tag in the header of your website.

Add the Meta Tag in The Header Manually

I hope you know how to add custom code in the header and footer of your WordPress website. You can either use a plugin like Insert headers and footers or do it manually by editing the theme files.

Add this line in the header of your website and it will tell the search engines not index your website. But people may get confused with the file editing.

Edit the Robots.txt File to Block Web Crawlers

The robots.txt file is just a text file which is used to handle the crawling of your website. You can set the folders and directories you want the web crawlers to crawl.

You can disallow all the bots at once. This file can only request the web crawlers not to crawl your website. The rest is up to those.

Just create a text file and add the shown code.

Save the file and you’re done.

But there are still chances that your website can get crawled because not all the bots read your written rules.

For the complete blocking, you should password protect your whole website. It can be done just like you password protected your WordPress admin directory.

Stop Search Engine from Crawling Your WordPress Website by Password Protecting the Website Directory

As you all know the data of your website is present in the public_html directory. If you have moved your website to somewhere else, that’s a totally different situation.

There are a few steps you should follow.

Step 1: – Login to your cPanel account provided by the web hosting and seek for the security section.

Click on the “Password Protect Directories” icon and it will take you to a new page.

Step 2: – You will see many directories on the list. Here, you need to password protect the public_html directory because the website data is present in this directory only.

Click on the directory and you will see many fields to fill.

Step 3: – Add the username, the password which will be used when anyone would try to open your website. Without these login credentials, no one will be able to crawl your website, not even the bots.

Save the settings and you’re all set.

You have successfully password protected your entire WordPress website. This is the best way to stop search engine from crawling your WordPress website.

Many people get confused with the above-mentioned concepts. But if you can access your WordPress admin panel then you can use the first method and discourage the search engine to index your site.

That’s the simplest method. I hope you have learned the things you desired. If you still have any questions, feel free to drop a comment.

You can also connect with us on Twitter, LinkedIn, and Facebook.